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Abstract 

We give in this paper a convergence result concerning parallel synchronous 
algorithm for nonlinear fixed point problems with respect to the euclidian 
norm in R". We then apply this result to some problems related to convex 
analysis like minimization of functionals, calculus of saddle point, convex 
programming... 
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1 Introduction. 

This study is motivated by the paper of Bahi[3] where he has given a conver- 
gence result concerning parallel synchronous algorithm for linear fixed point 
problems using nonexpansive linear mappings with respect to a weighted 
maximum norm. Our goal is to extend this result to a nonlinear fixed point 
problems, 

F(x*)=x* (1) 

with respect to the euclidian norm, where F : lR n — > M n is a nonlinear 
operator. 

Section [2] is devoted to a brief description of asynchronous parallel algorithm. 
In section [3] we prove the main result concerning the convergence of the 
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general algorithm in the synchronous case to a fixed point of a nonlinear 
operator from R™ to R™. A particular case of this algorithm (Algorithm of 
Jacobi) is applied in section H] to the operator F — (I + T) _1 which is called 
the proximal mapping associated with the maximal monotone operator T 
(see Rockafellar[9]). 



2 Preliminaries on asynchronous algorithms. 

Asynchronous algorithms are used in the parallel treatment of problems tak- 
ing in consideration the interaction of several processors. Write R n as the 

a a 

product Y\ where a G N — {0} and n = Y2 n i- All vectors x G R™ consid- 

i=l i=l 

ered in this study are splitted in the form x = (xi, ■■■,x a ) where Xj G R ni . Let 
R™ 4 be equipped with the inner product (., .) { and the associated norm ||.||,. = 

a 

(., .)/ . R n will be equipped with the inner product (x, y) = Yl ( x ii Vi)i where 

i=i 

a 

i,i/6 W 1 and the associated norm ||x|| = (x,x) = ) ■ It will be 



equipped also with the maximum norm defined by, 



i=l 



x l = max x,- 



l<j<a 



Define : 

J = {J(p)} peN a sequence of non empty sub sets of {1, a} and 
S = {(si(p), s a (p))} gN a sequence of N Q such that, 

• Wi G {1, a}, the subset {p G N, i G J{p)} is infinite. 

• Vz G {1, a} , Vp G N, Si(p) <p. 

• Wi G {1, a} , lim Si(p) = oo. 

p— >oo 

Consider an operator F = (iq, F a ) : lR n — > lR n and define the asyn- 
chronous algorithm associated with F by, 

' x° = (x°,...,x°) G R n 



x? +1 



xf if i£J (p) 

F^M,...,^) if ieJ{p) (2) 



i — 1, a 



{ P = 0,1,.. 

(see Bahi and al.pQ, El Tarazijl]). It will be denoted by (F, x°, J, S). This al- 
gorithm describes the behaviour of iterative process executed asynchronously 
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on a parallel computer with a processors. At each iteration p + 1, the i th 
processor computes x P+1 by using (j2J) (Bahi[2]). 

J(p) is the subset of the indexes of the components updated at the p th step. 
p — Si(p) is the delay due to the i th processor when it computes the i th block 
at the p th iteration. 

If we take St(p) = pWi G {1, a}, then describes synchronous algorithm 
(without delay). During each iteration, every processor executes a number of 
computations that depend on the results of the computations of other pro- 
cessors in the previous iteration. Within an iteration, each processor does 
not interact with other processors, all interactions takes place at the end of 
iterations (Bahi[3]). 
If we take 

Si(p)=p Vp G N, Vz G {1, a} 

J(p) = {l,...,a} VpeN 

then ([2D describes the algorithm of Jacobi. 
If we take 

Si{p)=p Mp G N, Vz G {1, a} 

J(p) — p + 1 (mod a) Vp G N 

then (|2|) describes the algorithm of Gauss-Seidel. 

For more details about asynchronous algorithms see pQ, [2j, [H] and [3]. 
In the following theorem, Bahi[3] has shown the convergence of the sequence 
{x p } defined by (j2J) in the synchronous linear case, i.e F is a linear operator 
and Si(p) =p, \/p G {1, a}. 

Theorem 1 Consider {T p } peN a sequence of matrices in W nxn . Suppose 
(ho) 3 a subsequence {pk}km such that J(vk) = {1, •••,«} , 
(hi) 37 ^> C0, \/p G N, T p is nonexpansivq^ with respect to a weighted maxi- 
mum norm ||.|| 007 defined by 

x G W\ II 2 II „ = max ^ X ^ 1 

00,7 \<i<a 7j 

(h 2 ) {T p } peN converges to a matrix Q which is paracontracting with respect 
to the norm ||.H . 

II 1100,7 

(h 3 ) Vp G N, N(J -Q)Q N(I - T p ) (N denotes the null space), 
then 



1. Vr G W 1 the sequence {x p } gN is convergent 



m 



1 7^0 means ji > Vi £ {1, a} 

2 A matrice A £ R nxra is said to be nonexpansive with respect to the norm ||.|| if Va; S 
! , \\Ax\\ < \\x\\. A is said to be paracontracting if Vx £ R n , x ^ Ax <=> ||Ax|| < ||x||. 
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2. lim x p = x* G M{I - Q) 

p— >oo 

Proof. See Bahi[3]. ■ 

Remark 1 The hypothesis (ho) means that the processors are synchronized 
and all the components are infinitely updated at the same iteration. This 
subsequence can be chosen by the programmer. 



3 Convergence of the general algorithm. 

We establish in this section the convergence of the general parallel syn- 
chronous algorithm to a fixed point of a nonlinear operator F : W 1 — > M n 
with respect to the euclidian norm defined in section [21 We recall that an 
operator F from R™ to R n is said to be nonexpansive with respect to a norm 
11-11 if, 

Wx,yeR n , \\F(x) - F(y)\\ < \\x - y\\ 



Theorem 2 Suppose 

(ho) 3 a subsequence {pk}km such that J(pk) = {1, ■■■,ac} 
(hi) 3u G R n , F(u) = u 

(h 2 ) Vx, y G M", \\F(x) - F(y)\\ < \\x - yW^ 

(h 3 ) Vx, y G M", \\F(x) - F(yW < (F(x) - F(y),x - y) 

Then any parallel synchronousQalgorithm defined by (dj) associated with the 

operator F converges to a fixed point x* of F. 

Proof. 

(i) We prove first that the sequence {x p } pGN is bounded. 
Vz G {1, a} we have, 
either i J(p), so 

II p+i II ii p n 

— Ui\\. = \\Xi — UiW- 

?, H o 7, L ?. 



< \\ xV - U \\oo 



or % G J(p), so 



K +1 -<4 = mixn - F(u)i 

< \\F( X p) - Fiu)^ 

< lk P -«lloo (by (h 2 )) 



In this case Si(p) = p Mi G {1, a} Vp G N. 
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Vi G {1, a} , \\x P+1 — Ui\\ < \\x p — u 
Vp G N, ||x p+1 -u\\ < \\x p -u" 



oo 



SO 

then 
hence 

V» G N, ||x p - dl <: lk° - u\\ 

r 'II ii oo — II lloo 

this proves that the sequence {x p } p€N is bounded with respect the max- 
imum norm and then it's bounded with respect the euclidian norm . 

[ii) As the sequence {x Pk } keN is bounded ({Pfc} feGN is defined by (h )), it 
contains a subsequence noted also {x Pk } keN which is convergent in IR n 
to an x*. We show that x* is a fixed point of F. For it, we consider the 
sequence {y p = x p — F(x p )} p€ n and prove that lim y Pk = 0. 

fc^oo 

|2 



\ x vk_ u \\* = \\ y Pk + F(x pk ) - u\ 



\y Pk f + \\F(x Pk ) -u\\ 2 + 2(F(x Pk ) -u,y Pk ) 



however 



(F(x Pk ) - u, y Pk ) = (F(x Pk ) - F(u), x Pk - F(x Pk )) 

= (F(x Pk ) - F(u), [x Pk - F(u)] - [F(x Pk ) - F(u)}} 

= (F(x Pk ) - F(u),x Pk -u) - \\F(x Pk ) - F(u)f 

> 0(by(h 3 )) 

so, 

\\y Pk f < \\x Pk - u\\ 2 - \\F(x Pk ) - u\\ 2 

= \\x Pk — u\\ 2 — \\x Pk+1 — u\\ 2 (by (ho)) 

However, in (TJ) we have shown in particular that the sequence {\\x p — uH^} gN 
is decreasing (and it's positive), it's therefore convergent, then the se- 
quence {\\x p — u\\} peN is also convergent, so 

lim — u\\ = lim \\x Pk — u|| 
= lim \\x Pk+1 -u\\ 

k— >oo 



\X - U\ 



and so 



which implies that 



lim \\y Pk \ 

k— »oo 



x* -F(x*) 
that is x* is a fixed point of F. 
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(Hi) We prove as in (Tj| that the sequence {\\x p — ^*|| 00 } p6N is convergent, so 
lim \\x p — x*\\ = lim \\x Pk — x*\\ = 

Which proves that x p — > x* with respect to the uniform norm H-H^- ■ 

Remark 2 We have used the hypothesis (h 2 ) to prove that the sequence 
{x p } p&N is bounded. In the case of the parallel algorithm of Jacobi where 
J(p) = {l,...,a} Wp G N, we don't need this hypothesis, since in this case 
x p+i _ _p( x P) \/p ^ an d use (ji^ t obtain 

\\x p+l - u || = \\F(x p ) - F(u)\\ < \\x p - u\\ , 
hence the corollary, 

Corollary 3 Under the hypotheses (h\), (/13) and 

(h' )\/ P eN, J(p) = {i, ...,«} 

The parallel Jacobi algorithm defined by 

/y>0 ( , y» ^ _ 

%Xj \ JU 1 - • • • . JU 



1) "•) ^OLj 



i = 1, .... a 



p=l,2... 

converges in W 1 to an x* fixed point of F. 



4 Applications. 

4.1 Solutions of maximal monotone operators. 

In this section, we apply the parallel Jacobi algorithm to the proximal map- 
ping F — (I + T) _1 associated with the maximal monotone operator T. We 
give first a general result concerning the maximal monotone operators. Such 
operators have been studied extensively because of their role in convex anal- 
ysis (minimization of functionals, min-max problems, convex programming, 
...) and certain partial differential equations (Rockafellar[9]). 
Let T be a multivalued maximal monotone operator defined from IR n to R n . 
A fundamental problem is to determine an x* in R™ satisfying G Tx* which 
will be called a solution of the operator T. 

Theorem 4 Let T be a multivalued maximal monotone operator such that 
T _1 7^ 0. Then every parallel Jacobi algorithm associated with the single- 
valued mapping F = (I + T) _1 converges in ~R n to an x* solution of the 
problem G Tx. 
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Proof. 

G Tx xe(I + T)x 

x = (I + T)- 1 x (4) 
•<=>- x = Fx 

Thus, the solutions of T are the fixed points of F, so the condition T _1 7^ <fi 
implies the existence of a fixed point u of MP. It remains to show that F 
verifies the condition (/t 3 ) and apply Corollary [31 Consider x % G MP (i = 1, 2) 
and put y J = Fx 1 then x* G y % + T?/ 1 or x* — y % G Ty\ As T is monotone we 
have ((x 1 — y 1 ) — (x 2 — y 2 \y x — y 2 ) > and therefore (x 1 — x 2 , y 1 — y 2 ) — 
Wy 1 ~ y 2 \\ 2 > which implies WFx 1 — Fx 2 || 2 < (Fx 1 — Fx 2 ,x l — x 2 ) ■ 

4.2 Minimization of functional. 

Corollary 5 Let f : M n — > MU{oo} 6e a lower semicontinuous convex func- 
tion which is proper (i.e not identically +00). Suppose that the minimization 
problem mm. fix) has a solution. Then any parallel Jacobi algorithm associ- 

R" 

ated with the single-valued mapping F = (I + df)^ 1 converges to a minimizer 
of f in M n . 

Proof. Since in this case the subdifferential df is maximal monotone. More- 
over the minimizers of / are the solutions of df. We then apply Theorem H] 
to df. m 

4.3 Saddle point. 

In this paragraph, we apply Theorem H] to calculate a saddle point of func- 
tional L : M n x MP — ■> [—00, +00]. Recall that a saddle point of L is an 
element (x*,y*) ofM n x M p satisfying 

L(x\y) < L(x*,y*) < L(x,y*), V(ar,y) G M n x M p 

which is equivalent to 

L(x*,y*) = inf L(x,y*) = sup L(x*,y) 



Suppose that L(x,y) is convex lower semicontinuous in x G M n and concave 
upper semicontinuous in y G MP. Such functionals are called saddle functions 
in the terminology of Rockafellar[6]. Let Tl be a multifunction defined in 
M n x W by 



(u,v) G T L {x,y) 



L(x, y') + (2/ -y,v) < L(x, y) < L(x', y) - (x' - x, u) 
V(x', y') e 1" x l p 
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If L is proper and closed in a certain general sense, then Tl is maximal 
monotone; see Rockafellar [BUT] . In this case the global saddle points of L 
(with respect to minimizing in x and maximizing in y) are the elements 
(x, y) solutions of the problem (0,0) G Ti,(x,y). That is 

(0, 0) G T L (x*, y*) •<=>- (x*,y*) = arg mm max L(x,y) 

iei" yeRp 

We can then apply Theorem H] to the operator Tl so, 

Corollary 6 Let L be a proper saddle function from M n x MP into [— oo, +oo] 
having a saddle point. Then any parallel Jacobi algorithm associated with the 
single-valued mapping F = (I + Tl)' 1 from MP x MP into M n x MP converges 
to a saddle point of L. 



4.4 Convex programming. 

We consider now the convex programming problem, 

/ Minf (x), xeW 1 
1 } \ fi(x) < 0, (1 < % < m) {0> 

where fi : M n — > M (0 < % < m) is lower semicontinuous convex functions. 
This problem can be reduced to an unconstrained one by mens of the La- 
grangian, 



L(x,y) = f (x) + ^y l f l 



i=l 

where x G MP and y G (M + ) m . We observe that L is a saddle function in the 
sense of [6jp. 363], due to the assumptions of convexity and continuity The 
dual problem associated with (P) is, 

f Max { 90 (y) = ig n L(x,y)} 
\ y G (R + ) m 

If (x*, y*) is a saddle point of the Lagrangian L then x* is an optimal solution 
of the primal problem (P) and y* is an optimal solution of the dual problem 
(D). 

Let dL(x,y) the sub differential of L at (x,y) G M n x M p , be defined as the 
set of vectors (u, v) G M n x MP satisfying 

V(x\ y') EM n x M p , L{x, y') - (y' -y,v)< L{x, y) < L{x', y) - (x' - x, u) 
(see LuquejS] and Rockafellar 

Then the operator Tl : (x,y) — > {(u, v) : (u,—v) G dL(x,y)} is maximal 
monotone (Rockafellar P, Cor. 37.5.2]), so we apply Theorem H] to Tl- 
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Corollary 7 Suppose that the convex programming (P) defined by $5$ has a 
solution. Then any parallel Jacobi algorithm associated with the single-valued 
mapping F = (I + Tl)~ 1 from M. n x M. p to M. n x M. p converges to a saddle point 
(x*,y*) of L, and so x* is a solution of the primal (P) and y* a solution of 
the dual (D). 



4.5 Variational inequality. 

A simple formulation of the variational inequality problem is to find an x* G 
M™ satisfying 

(Ax*,x — x*) > Vx G M" (7) 

where A : IR n — ► W l is a single-valued monotone and maximal operatoiQ. 
Which is equivalent to find an x* G M n such that 

G Ax* + N(x*) 

where N(x) is the normal cone to M. n at x defined by (see Rockafellar pi9"] ) . 

N(x) = {ye R n : (y, x - z) > Vz G R n } 

Rockafellar [H] has considered the multifunction T defined in R n by 

Tx = Ax + N(x) (8) 

and shown in [5] that T is maximal monotone. The relation G Tx* is so 
that reduced to —Ax* G N(x*) or (— Ax*,x* — z) > Vz G K" which is the 
variational inequality (JTj). Therefore the solutions of the operator T (defined 
by (jHJ)) are exactly the solutions of the variational inequality (JTJ). By using 
Theorem H] we can write 

Corollary 8 Let A : IR n — » IR n be a single-valued monotone and hemicon- 
tinuous operator such that the problem |7p has a solution, then any parallel 
Jacobi algorithm associated with the single-valued mapping F = (/ + T) _1 
where T is defined by (Ejj converges to x* solution of the problem |?|). 
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